Week 6 of 12 · The Craft

Phase 1 Complete

Six weeks, one story: from a single rubric to an honest eval — and the half-track that turned you from labeler to lead

Day 30 ~60 minutes Review

Day 30 of 66

You're at the halfway mark — look at what you can do

Thirty days ago you produced human data. Now you can design and measure it. Phase 1 — The Craft — is done, and it wasn't six unrelated topics. It was one arc, told in order. Say it back without looking.

The Phase 1 arc, as one story

Week 2 — Rubric. You wrote the document that defines "good." Everything downstream inherits its quality.
Week 3 — Agreement. You measured whether raters apply the rubric consistently — kappa, gold sets, sampling — turning "I think it's good" into a number.
Week 4 — Preference. You turned rubric-scored outputs into chosen/rejected pairs that train the reward model.
Week 5 — Multimodal. You extended all of it to image and video — your edge — where the output isn't text and the hard dimensions are yours to define.
Week 6 — Eval. You built the instrument that measures the model to decide ship / no-ship, with a debiased judge and an honest, un-gameable design.

The through-line

Rubric defines good → agreement proves raters apply it consistently → preference data trains the model toward good → multimodal makes it real for your domain → eval honestly measures whether you got there. Each week's artifact feeds the next. That chain is the human-data operation. You've now built every link.

The memo: three biases, three mitigations

This is the piece of your capstone that proves you understand evaluation, not just ran it. Write it short and exact — three bullets, each naming a bias from your reading and the specific control your design uses against it.

Model memo — write yours like this

Position bias (the judge favors a slot): my harness runs every comparison twice with the order swapped and counts only self-consistent verdicts, discarding the flips. Verbosity bias (favors longer answers): my eval controls for length so the judge can't reward padding over correctness. Self-enhancement bias (a model favors its own family): I never let Grok judge Grok — I use a neutral third-party judge. And before trusting any of it, I validated the judge against human labels and reported the agreement rate.

Self-quiz (no notes)

Answer out loud or in writing. Any shaky answer means reread that day before Phase 2.

A teammate wants to add some eval prompts to the SFT set "since they're good examples." What do you say, and what's the word for the danger?
Two models are being compared for launch. Pairwise or Likert — and why?
A judge reports the new model wins 90% of comparisons. Name two checks you run before you believe it.
Your eval scores keep rising but users complain more. What law describes this, and what likely went wrong?
In one sentence: why does an eval set have to be held out?

Answer key (peek after you try)

1) Refuse — that's contamination; the eval can no longer measure honestly because the model will have seen the answers. 2) Pairwise: more reliable (relative judgments are consistent) and the ship question is inherently relative. 3) Check the judge's self-consistency under order swap and its agreement with human labels; also check the eval set for contamination. 4) Goodhart's law — the metric became the target and decoupled from real quality; the eval is flattering-but-broken (too narrow, biased judge, or contaminated). 5) Because evaluation measures the model; if it's trained on, the measurement is no longer honest.

Close Phase 1

Wrap-up

Before Week 7 (Phase 2 begins)

Self-quiz answers all solid (or reread the weak day)
The three-biases memo written — one bullet per bias, each with your design's mitigation
Phase 1 told as one narrative: rubric → agreement → preference → multimodal → eval
The weakest link in your own eval identified, with how you'd fix it

Up next — Phase 2: The Lead

Phase 1 made you fluent in the craft of human data. Phase 2 makes you the person who runs the operation: the data analysis stack (Week 7), annotation pipelines and tooling (Week 8), quality ops at scale, leading annotator teams, safety and red-teaming, and a full capstone program. The skills compound — every Phase 2 artifact reuses one you built here.

The lead move

An IC can do each piece — write a rubric, score agreement, build a pair, run a judge. A lead can tell the whole story as one chain: rubric defines good → agreement proves raters apply it → preference data trains toward it → multimodal makes it real for the domain → eval honestly measures whether you got there. Owning the arc, not the task, is the altitude jump.

Say this in an interview: "I don't think of human data as separate jobs — it's one operation. The rubric defines good, agreement proves the team applies it consistently, preference data trains the reward model toward it, and a held-out, debiased eval measures whether we actually shipped a better model. Each artifact feeds the next, and I can own any link in that chain."

Phase 1 Takeaways

You can author a rubric, prove raters apply it, build preference data, extend it all to multimodal, and measure the result with an honest eval.
You know evaluation is its own discipline — held-out sets, contamination, pairwise vs Likert, and the shipping gate.
You can name and mitigate the three LLM-judge biases, and you validate a judge against humans before trusting it.
You can spot a flattering eval (Goodhart) before it ships a worse model — and you're halfway to the role.